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(54) Remote control method, server and recording medium 



(57) A window system of the kind used in personal 
compuiors is cpninhlo in a portable information terminal 
which docs noi lv<vc a sufficient processing capability 
by having input inlormation. such as a key input from an 
operator, gcncraicd in a portable information terminal 
210 sent to a rcoioic supporting server 240 where it ex- 
ecutes or simulates an arbitrary window system which 
is free from restriction of functions to generate a render- 
ing image The supporting server then converts the ren- 
dering imago to an image suitable to the portable infor- 
mation terminal and sends it back to the portable infor- 
mation terminal 2 1 0 as a rendering instruction. The port- 
able information terminal 210 receives the rendering in- 
struction of this optimised image for display. 
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Description 

This invention relates to a remote control method 
and. in particular, to an improved remote control method 
in which resources required (or a machine to be remote 
controlled are reduced. 

An OS (Operating System) providing a GUI (Graph- 
ic User Interlace) and capable of file processing has 
been so tar operable on a portable terminal such as a 
notebook PC. 

However, a certain degree of machine power was 
required for a portable terminal in order to run such OS. 
For example. Windows 95 (Windows 95 is a trademark 
of Microsoft) requires provision of a CPU which is equal 
to or above i486SX, a memory larger than 8f^B. a hard 
disk larger than 75 MB and a video display adapter hav- 
ing a resolution above the VGA. 

On the other hand, a ponable terminal, such as a 
smartphone, which has a small memory capacity and Is 
provided only with a telephone function and an output 
function of a simple display, etc., including •Datascope" 
of Kyocera and 'Pinocchio" of Panasonic, can only op- 
erate a special chip OS and can not use a general pur- 
pose OS. 

There have been several operating systems, such 
as the above chip OS. specialised for and operable sole- 
ly on a portable information terminal. 

However, they are accompanied with a problem that 
the function is limited due to the limited machine power, 
menrK>ry capacity and display capability and they are un- 
able to execute a high level application which is opera- 
ble on 6 general purpose OS as used in conventional 
PC's. 

On the other hand, a portable terminal which can 
operate a general purpose OS involves a problem that 
down-sizing would involve many restrictions and the 
cost would be increased. 

Further because the window system of the portable 
information terminal is specialised to such terminal, an 
application has to be often designed and developed 
solely for such window system and an application used 
in home or office can not be used in such terminal. 

In addition, while a portable information terminal is 
often carried along in a brief case, etc., by its nature, a 
danger was accompanied in which important data was 
lost by accident because data of an application special- 
ised for such portable information terminal is stored in 
a memory of such terminal. 

It is an object of this invention to provide a system 
which is capable of executing a general purpose OS as 
used in a conventional PC and a high level application 
operating on such OS in a machine having a limited re- 
sources. 

It is one of other objects in one mode of this inven- 
tion to provide a small sized portable terminal which can 
provide a high function with a low cost. 

It is one of other objects in one mode of this inven- 
tion to provide a portable information terminal which can 



execute an application used in home and office, etc. 

It IS one of other objects in one mode of this inven- 
tion to provide a ponable information terminal which has 
a low probability of losing data. 
S It is one of other objects in one mode of this inven- 
tion to reduce the work of installing a remote control soft- 
ware which was generated in the side of rennote control- 
ling terminal for enabling a consolidated control of a sys- 
tem. 

^0 It is one of other objects in one mode of this inven- 
tion to provide a system which is not affected by the type 
of a remote controlling terminal for realising a common 
operability. 

It is one of other objects in one mode of this inven- 
ts tion to reduce the usage of resources of a remote con- 
trolling terminal for reducing the hardware requirement 
to enable a general purpose OS and a general purpose 
application to be used. 

It is one of other objects in one mode of this inven- 
tion to provide a high speed remote control system 
which reduces the volume of communication data be- 
tween a remote controlling machine and a remote con- 
trolled machine in remote control operation. 

It is one of other objects in one mode of this inven- 
tion to provide a remote control system which can trans- 
mit data in an optimum form to a remote controlling ma- 
chine in remote control operation. 

It is one of other objects in one mode of this inven- 
tion to provide a remote control system which can simul- 
taneously support a plurality of remote controlling ma- 
chines in renrote control operation. 

Input information, such as a key input from an op- 
erator generated in a remote controlling machine is 
transmitted to a remote location supporting server which 
executes or simulates an arbitrary window system which 
is free from restriction of function to generate a render- 
ing image. The supporting server then converts the ren- 
dering image to an image which is suitable to the remote 
controlling machine and returns it to the remote control- 
ling machine as a rendering instruction. The remote con- 
trolling machine receives the rendering instruction of 
this optimised image and displays it. 

Because this window system actually operates on 
the supporting server, a window system virtiich is widely 
used in personal computers, etc., may be used as it is 
and also an application for personal computers may be 
used as it is. 

In one mode of this invention, a plurality of remote 
controlling machines can be simultaneously supported 
and, in response to processing capability specifying in- 
formation transmitted from individual remote controlling 
machine, a rendering instruction is transmitted after 
conversion toa form which is optimum to each individual 
remote controlling machine. 

In one of other modes of this invention, a method is 
provided for renrKJte controlling resources retained by a 
server existing on a network from a terminal which has 
an input device and a display device, conrprising the 
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Steps of: 

(a) accessing said server via said network. 

(b) transmitting a signal generated from the input 
device of said terminal to said server as input infor- 
mation. 

(c) receiving in said server side the input information 
transmitted from said terminal side and analysing 
the content of the input information, 

(d) generating first rendering Image data consisting 
of a first number of bits per picture element, 

(e) convening said first rendering image data to sec- 
ond rendering image data consisting of a second 
number of bits per picture element less than the first 
number of bits. 

(f ) transmitting a rendering Instruction including said 
second rendering image from said server side to 
said terminal side, 

(g) analysing the rendering instruction transmitted 
from the server side on said terminal side, and 

(h) generating image data to be displayed in a dis- 
play screen of said terminal according to the content 
of the analysed rendering Instruction. 

In one of other modes of this invention, a method is 
provided for remote controlling resources retained by a 
server existing on a network from a terminal which has 
an input device and a display device, comprising the 
steps of: 

(a) transmitting a connection request from a first ter- 
minal of said plurality of terminals to said server via 
said network 

(b) starting a shell process in said sen/er in re- 
sponse to the connection request from said first ter- 
minal. 

(c) transmitting processing capability specifying In- 
formation from said first terminal to said sen/er, 

(d) transmitting a signal generated from an inpul de- 
vice of said first terminal to said sen/er as Input tn- 
formatbn. 

(e) receiving the input information transmitted from 
said terminal side in said server side and analysing 
the content of the input information, 

(f) generating first rendering image data based on 
the content of the analysed Input information, 



(g) convening said first rendering image data to sec- 
ond rendering image data according to said 
processing capability specifying information. 

5 (h) transmitting a rendering instruction Including 
said second rendering image from said server side 
to said terminal side, 

(I) analysing the rendering instruction transmitted 
10 from the sen/er side in said terminal side. 

(j) generating image data to be displayed in a dis- 
play screen of said terminal according to the content 
of the analysed rendering instruction. 

In one of other modes of this invention, a method is 
provided for renriote controlling resources retained by a 
server existing on a network from a terminal which has 
an Input device and a display device, comprising the 
steps of; 

(a) receiving the input information transmitted from 
said terminal side, 

(b) analysing the input information. 

(c) generating first rendering image data consisting 
of a first number of bits per picture element based 
on the content of the analysed input Information, 

(d) converting said first rendering image data to sec- 
ond rendering image data consisting of a second 
number of bits per picture element less than the first 
number of bits. 

(e) transmitting a rendering Instruction including 
said second rendering image from said sen/er side 
to said terminal side. 

In one of other modes of this invention, a method is 
provided for renxDte controlling resources retained by a 
server existing on network from a terminal which has an 
input device and a display device, comprising the steps 
ot; 

(a) starting a shell process In response to a connec- 
tion request from a first terminal of said plurality of 
terminals, 

(b) receiving input information transmitted from said 
terminal side, 

(c) analysing the input information, 

(d) generating first rendering image data consisting 
of a first number of bits per picture element based 
on the content of the analysed input infornriation, 
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(e) converting said first rendering image data to sec- 
ond rendering image data consisting of a second 
number of bits per picture element less than the first 
number ol bits. 

(() transmuting a rendering instruction including said 
second rendering image from said server side to 
sard terminal side. 

In one ol other modes of this invention, a method is 
provided for remote controlling resources retained by a 
server existing on a network from terminal which has an 
input device and a display device, comprising the steps 
of. 

(a) iccciving a processing capability specifying in- 
lornrviicn iransmitied from said terminal. 

(b) icccrving input information transmitted from said 
tcrmirMl 

(c) dnritys ng the input information transmitted from 
saic Icmnal 

(d) gcrcrcjiing first rendering image data based on 
the coniont ol the analysed input information, 

(e) cons/crt ing said first rendering image data to sec- 
ond ronrtoring image data according to said 
processing cfipability specifying information. 

(I) transmitting a rendering instruction including said 
second rendering image from said server side to 
said terminal side. 

In one of ether modes of this invention, the process- 
ing capability specifying information is either one of; 

(a) maximum resolution information of a logicaf 
screen. 

(b) maximum resolution information of a physical 
screen, 

(c) bits per pcture element information. 

(d) colour palette information, 

(e) compiession/decompression capability infor- 
mation, of said terminal. 

In one of other modes of this invention, a method is 
provided for remote controtling resources retained by a 
server existing on a network from a terminal which has 
an input device and a display device, comprising the 
steps of; 

(a) starting a shell process in response to a connec- 



tion request from a first terminal of said plurality of 
termir^ls. 

(b) receiving processing capability specifying infor- 
5 mation transmitted from said terminal side, 

(c) receiving input information transmitted from said 
terminal side. 

10 (d) analysing the input information, 

(e) generating first rendering image data based on 
the content of the analysed input information, 

IS (f) converting said first rendering image data to sec- 
ond rendering image data according to said 
processing capability specifying information, 

(g) transmitting a rendering instruction including 
20 said second rendering image from said server side 
to said terminal side. 

In one of other modes of this invention, a system is 
provided which comprises a server existing on a net- 
2S work, and a terminal having an input device and a dis- 
play device capable of remote controlling resources of 
said server retained by said server, comprising; 

(a) a terminal; 

30 

(a-1) accessing said server via said network. 

(a-2) transmitting a signal generated from the 
input device of said terminal to said server as 
35 input information, 

(a-3) analysing a rendering instruction trans- 
mitted from the server side, 

40 (a-4) generating image data to be displayed in 

a display screen of said terminal according to 
the content of the analysed rendering instruc- 
tion, and 

45 (b) a server; 

(b-1) receiving input information transmitted 
from said terminal side and analysing the con- 
tent of the input information, 

so 

(b-2) generating first rendering inr^age data con- 
sisting of a first number of bits per picture ele- 
ment based on the content of the analysed in- 
put information, 

55 

(b-3) converting said first rendering image data 
to second rendering image data consisting of a 
second number of bits per picture element less 
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than the first number of bits, 

(b-4) transmitting a rendering instruction in- 
cluding said second rendering image from said 
server side to said terminal side. 

In one of other modes of this invention, a system is 
provided which comprises a server existing on a net- 
work, and a plurality of temiinals having an input device 
and a display device capable of remote controlling re- 
sources of said server retained by said server, compris- 
ing; 

(a) a plurality of terminals; 

(a-1 ) transmitting a connection request to said 
server via said network. 

(a-2) transmining processing capability speci- 
fying information to said server, 

(a-3) transmitting a signal generated from the 
input device to said server as input information, 

(a-4) analysing a rendering instruction trans- 
mitted from the server side. 

(a-5) generating image data to be displayed in 
a display screen according to the content of the 
analysed rendering instruction, and 

(b) a server: 

(b-1) starting a shell process in response to a 
connection request from the terminal side. 

(b-2) receiving input information transmitted 
from said terminal side and analysing the con- 
tent of the input information, 

(b-3) generating first rendering image data 
based on the content of the analysed input in- 
formation, 

(b-4) convening said first rendering image data 
to second rendering image data according to 
said processing capability specifying Informa- 
tion, 

(b-5) transmitting a rendering instruction in- 
cluding said second rendering image from said 
server side to said terminal side. 

In one of other modes of this Invention, a server Is 
provided which exists on a network and is remote con- 
trolled from a terminal which has an Input device and a 
display device, comprising; 



(a) an input infornnation receiving part for receiving 
input information transmitted from said terminal 
side, 

s (b) an input information analysing part for analvsing 
the input information. 

(c) an image rendering engine for generating first 
rendering image data consisting of a first number of 

10 bits per picture element based on the content of the 
analysed input information, 

(d) a rendering instruction monitoring part for con- 
verting said first rendering image data to second 

15 rendering image data consisting of a second 
number of bits per picture element less than the first 
number of bits, 

(e) a rendering instruction transmitting part for 
20 transmitting a rendering instruction Including said 

second rendering image from said server side to 
said terminal side. 

In one of other modes of this invention, a server is 
25 provided which exists on a network and is remote con- 
trolled from a terminal which has an input device and a 
display device, comprising: 

(a) an input infornnation receiving part; 

30 

(a-1) starting a shell process in response to a 
connection request from a first terminal of said 
plurality of terminals and, 

35 (a-2) receiving input information transmitted 

from said first terminal, 

(b) an input information analvsing part for analysing 
the input information. 

40 

(c) an image rendering engine for generating 
first rendering image data consisting of a first 
number of bits per picture element based on the 
content of the analysed input information, 

45 

(d) a rendering instruction monitoring part for 
convening said first rendering innage data to 
second rendering image data consisting of a 
second number of bits per picture element less 

so than the first number of bits, 

(e) a rendering Instruction transmitting part for 
transmitting a rendering instruction Including 
said second rendering image from said server 

55 side to said terminal side. 

In one of other modes of this invention, a server Is 
provided which exists on a network and Is remote con- 
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Ifollcd Ifom a terminal which has an input device and a 
di^plny device comprising; 

(i) rin input information receiving pan;- 

(H- 1 ) receiving processing capability specifying 
iniornnation transmitted from said terminal, and 

{H'2) receiving input information transmitted 
Irom said first terminal, 

(b) an input infomnation analysing part for analysing 
tic nput information, 

c t an imnge rendering engine for generating first 
jcooernq image data based on the content of the 
^rLifysoo npjt information, 

f3. »(vxx?nng instruction monitoring part for con- 
vcii*>'j viO Nisi rendering image data to second 
i^i^X'ttig HiiriL^c data according to said processing 
Crti>it/<<?y specifying information, 

id .1 fcrocimg instruction transmitting part for 
iMnon^tinq .j rendering instruction including said 
toconc! fcndonng image from said server side to 
S^»C lo mi'Vil Side 

It mr Af nihof modes of this invention, a server is 
proviriixi wwiirri exists on a network and is remote con- 
trolled ticrr ^ icrminal which has an input device and a 
display dovKTC comprising: 

(a) an npul information receiving part; 

(^•1 1 starting a shell process in response to a 
connccton request from a first terminal of said 
piLfrii ty of terminals and, 

in 2) receiving processing capability specifying 
inlornvition transmitted from said first terminal, 

ia-3) receiving input information transmitted 
Irom said first terminal, 

(b) an npui inlormation analysing part for analysing 
the input information. 



transmitting a rendering instruction including said 
second rendering image from said server side to 
said first terminal side. 

s In one of other modes of this invention, a recording 
medium is provided which stores therein a remote con- 
trolled program executed by said server for remote con- 
trolling resources retained by a server existing on a net- 
work from a terminal which has an input device and a 

^0 display device, comprising; 

(a) a program code instructing said server to receive 
input information transmitted from said terminal 
side, 

75 

(b) a program code instructing said server to ana- 
lyse the input information, 

(c) a program code instructing said server to gen- 
^0 erate first rendering image data consisting of a first 

number of bits per picture element, 

(d) a program code instructing said server to con- 
vert said first rendering innage data to second ren- 

25 dering image data consisting of a second number 
of bits per picture element less than the first number 
of bits, 

(e) a program code instructing said server to trans- 
30 mit a rendering instruction including said second 

rendering image from said server side to said ter- 
minal side. 

In one of other modes of this invention, a recording 
35 medium is provided which stores therein a remote con- 
trolled program executed by said server for remote con- 
trolling resources retained by a server existing on a net- 
work from a terminal which has an input device and a 
display device, comprising: 

40 

(a) a program code instructing said server to start 
a shell process in response to a connection request 
from a first terminal of said plurality of terminals, 

^5 (b) a program code instructing said server to receive 
input information transmitted from said first terminal 
side, 



(c) an iirvige rendering engine for generating first 
render r»g image data based on the content of the 50 
analysed input information, 

(d) a rendering instruction monitoring part for con- 
verting said first rendering image data to second 
rendering inr>age data according to said processing 55 
capability specifying information, 

(e) a rendering instruction transmitting part for 



(c) a program code instructing said server to ana- 
lyse the input information, 

(d) a program code instructing said server to gen- 
erate first rendering image data consisting of a first 
number of bits per picture element based on the 
content of the analysed input infornoation, 

(e) a program code instructing said server to con- 
vert said first rendering inr^ge data to second ren- 
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dering image data consisting of a second nunnber 
of bits per picture element less than the first number 
of bits. 

(f) a program code instructing said server to trans- 
mit a rendering instruction including said second 
rendering image from said server side to said first 
terminal side. 

In one of other modes of this invention, a recording 
medium is provided which stores therein a remote con- 
trolled program executed by said server for remote con- 
trolling resources retained by a server existing on a net- 
work from a terminal which has an input device and a 
disptny device, comprising; 

program code instructing said server to receive 
processing capability specifying information trans- 
minxJ from said terminal, 

(b I rt program code instructing said server to receive 
input information transmitted from said terminal, 

\Z) .1 p'ogram code instructing said server to ana- 
tysc the rrput information. 

(dj H program code instructing said server to gen- 
crHic lirsi rendering Image data based on the con- 
i.'^ni of iho analysed input Information, 

(c) program code instructing said server to con- 
vert sHtd first rendering image data to second ren- 
dering image data according to said processing ca- 
pability specifying information. 

(I) program code instructing said server to trans- 
mit a rendering instruction including said second 
rcndcroig image from said server side to said first 
terminal side 

In one of other modes of this invention, a recording 
medium is provided which stores therein a remote con- 
trolled program executed by said server for remote con- 
trolling resources retained by a server existing on a net- 
work from a terminal which has an input device and a 
display device, comprising; 

(a) a program code instructing said server to start 
a shell process in response to a connection request 
from a first terminal of said plurality of terminals, 

(b) a program code instructing said server to receive 
processing capability specifying Information trans- 
mitted from said first terminal. 

(c) a program code instructing said server to receive 
input information transmitted from said first termi- 
nal. 



(d) a program code Instructing said sen/er to ana- 
lyse the Input Information, 

(e) a program code instructing said server to gen- 
s erate first rendering image data based on the con- 
tent of the analysed input Information, 

(f) a program code instructing said server to convert 
said first rendering image data to second rendering 

10 image data according to said processing capability 
specifying information. 

(g) a program code instructing said server to trans- 
mit a rendering instruction including said second 

15 rendering image from said server side to said first 
terminal side. 

In one of other modes of this invention, a method is 
provided for specifying a display region of size of (wx, 

20 wy) lo be displayed on a display screen from image data 
constituting a logical screen having a size of (sx, sy) 
stored in a portable Information terminal, characterised 
in that the region of the size of (wx. wy) is specified as 
a display region starling from a starting point which Is 

25 defined as; 

ox = px(sx - wx)/((sx - 1) 

30 

oy = py(sy- wy)/(sy- 1) 

where (px, py) is a co-ordinate position of a mouse point- 
er of the portable Information terminal. 
35 How the invention may be carried out will now be 
described with reference to the accompanying drawings 
in which; 

Fig. 1 is a block diagram of an embodiment of the 
hardware configuration of a remote control machine or 
a supporting server 

Fig. 2 is a diagram showing an embodiment of this 
invention in operational condition. 

Fig. 3 Is a block diagram showing the structure of 
portable information terminal and a supporting sender in 
a preferred embodiment of this invention. 

Fig. 4 is a flow chart showing the operational con- 
dition of a portable information terminal in a preferred 
embodiment of this Invention. 

Fig. 5 is a flow chart showing the operational con- 
dition of portable information terminal in a preferred em- 
bodiment of this invention. 

Fig. 6 Is a flow chart showing the operational con- 
dition of supporting server in a preferred embodiment of 
this invention. 

Fig. 7 Is a flow chart showing the operational con- 
dition of supporting server in a preferred embodiment of 
this invention. 

Fig. 8 is a flow chart showing the operational con- 
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dilion of a supporting server in a preferred embodiment 
of this invention. 

Fig. 9 is a flow chart showing the operational con- 
dition of a supporting server in a preferred embodiment 
of this invention. 

Fig. 10 is a flowchart showing the operational con- 
dition of a supporting server in a preferred embodiment 
of this invention. 

Fig. 11 is a schennatic diagram showing the opera- 
tional condition of a portable information terminal and a 
supporting server in a preferred embodiment of this in- 
vention. 

Fig. 1 2 is a diagram showing an embodiment of data 
conversion performed by an input infornnation generat- 
ing part in a preferred embodiment of this invention. 

Fig. 1 3 is a diagram showing an embodiment of in- 
put information and data in a preferred embodiment of 
this invention. 

Fig. 1 4 is a diagram showing an embodiment of data 
conversion performed by an input information analysing 
part in a preferred embodiment of this invention. 

Fig. 15 is a diagram showing an embodiment of a 
rendering instruction data in a preferred embodiment of 
this invention. 

Fig. 16 is a diagram showing an example of image 
conversion in a preferred embodiment of this invention. 

Fig. 17 is a diagram showing an example of image 
conversion in a preferred embodiment of this invention. 

The embodiment of this invention is now described 
with reference to the drawings hereunder. Referring to 
Fig. 1. an embodiment of hardware configuration of re- 
mote controlling machine (portable Information terminal 
210) as used in this invention is schematically shown. 
A remote controlling machine 100 comprises a central 
processing unit (CPU) 1 and a memory 4. The CPU 1 
and the memory 4 are connected to a hard disk device 
1 3, an auxiliary memory device, (or a flash ROM 31 ) via 
a bus 2. A floppy disk device (or a recording medium 
such as MO. CD-ROM 23. etc.) 20 is connected to the 
bus 2 via a floppy disk controller 19. 

A floppy disk (or a recording medium such as MO, 
CD-ROM. etc.) is inserted in the floppy disk device (or 
a recording medium drive device such as MO. CD-ROM, 
etc.) 20. The floppy disk, a hard disk device 13 (or a 
recording medium such as MO. CD-ROM 23. DVD) and 
a ROM 1 4 may record therein a computer program code 
tor providing instructions to the CPU 1 in co-operation 
with an operating system to practice this invention, 
which instructions are executed by being loaded in the 
memory 4. The computer program may be compressed 
or divided into a plurality of pieces for recording in a plu- 
rality of media. 

The remote controlling machine 210 may be further 
provided with a user intertace hardware and have a 
pointing device (mouse, track ball and joystick, etc.) 7, 
a keyboard 6 and a display 1 2 for presenting visual data 
to the user. It is also possible to connect a printer via a 
parallel port 16 and a modem via a serial port 15. The 



remote controlling machine 210 can be connected to a 
network via a serial port 15 and a modem or via a com- 
munication adapter 18 (Ethernet and token ring) for 
communication with other computers. 
s A speaker 23 receives an audio signal which is D/ 
A converted (digital/analog conversion) by an audio 
controller 21 via an amplifier 22 to output as a voice. The 
audio controller 21 can also A/D convert voice informa- 
tion received from a microphone 24 to take In voice In- 
10 formation external to the system into the system. 

It will be readily understood that the remote control- 
ling machine 210 of this invention can be practised with 
a conventbnal personal computer (PC), a workstation. 
a notebook PC, a palm top PC, a network computer, var- 
is lous home electric appliances such as a television in 
which a computer is implemented, a game machine hav- 
ing a communication function, and a communication ter- 
minal having a communication function such as a tele- 
phone device, a facsimile device, a portable telephone 
device, a PHS and an electronic notebook, or a combi- 
nation thereof. It should be understood that these com- 
ponents are exemplary given and it is not meant that all 
these components are necessarily indispensable to this 
invention. 

Particularly, as seen from the use of a smartphone. 
which has a telephone function, an output function of a 
display, etc., and a send/receive function of an electron- 
ic mail Including "Datascope* of Kyocera and "Pinoc- 
chio* of Panasonic, as the remote controlling machine 
210 in the preferred embodiment of this invention, the 
audio controller 21 . the amplifier 22. the speaker 23 and 
the microphone 24 required for processing a voice, the 
keyboard 6. the mouse 7 and the key board/mouse con- 
troller 5 which enable a direct input from the operator, 
the CRT 12, the display device 11 , the VRAM 9 and the 
VGA 8 for presenting visual data to the user, and various 
recording medium processing devices 1 9, 25, 27 are not 
necessarily required components. It is sufficient for the 
keyboard 6, the mouse 7 and the keyboard/mouse con- 
troller 5 to have input means for performing a remote 
operation practically 

Each component of the remote controlling machine 
210 and various modifications Including combining a 
plurality of machines and distributing functions to each 
to practice this invention should be readily obvious to 
those skilled in the an and such modifications are in- 
cluded in the concept of this Invention. 

The supporting server 240 used in this invention 
may be also implemented by the hardware configuration 
shown in Fig. 1 in the same manner as the remote con- 
trolling nr^chine 210. 

In other words, it is also sufficient for the supporting 
server 240 to be provided with a function of receiving 
transmitted input information and converting It to a ren- 
dering instruction for sending, and a function of storing 
remote controlling and remote controlled softwares and 
sending the remote controlling software, so that it should 
be readily understood that the server 240 may be also 
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implemented by a communication terminal having a 
communic;^lion (unction such as a conventional person- 
al computer (PC), a workstation and a notebook PC, or 
a combination thereof. 

Hov/cvcr these components are exemplary given 
and It IS nci intended that all of those components are 
nccessririly indispensable components of this invention. 
Particularly because the hardware configuration de- 
scribed here IS required for operating a remote server 
(supporting server), the audio controller 21 , the amplifier 
22 the spcHkcr 23 and the microphone 24 required for 
processing a voice are not the indispensable compo- 
nents Also the keyboard 6, the mouse 7 and the key- 
boardnrousc coriroller 5 enabling direct input from the 
opcratoi arc not the indispensable components when 
the server 6 soic»y remote controlled. 

Ihc opcfHiinq system in the side of the remote con- 
trolling r-vichirc 2 10 may be implemented by one which 
supports GUI mu ti-window environment as a standard, 
such as WtfiOowb NT (d trademark of fy/licrosoft), Win- 
dows 95 M tirtUcirwiik of Microsoft), Windows 3.x (a 
tradenvif K oi WicrDsoli) OS/2 ( a trademark of IBM), Ma- 
cOS (a tfrtdcrrrtfk oi Apolc). and X-WINDOW system (a 
tradenrvifk o» Mr» on AIX (a trademark of IBM), one 
which IS in cr\ir.;Cicr b»'isc environment, such as PC- 
DOS (a ifHdcm.irk ol IBM) and MS-DOS ( a trademark 
of MicrosoC) ^ rc^l timo OS. such as OS/Open (a trade- 
mark of IBM( nn6 VxWorks{a trademark of Wind River 
Systems Inr i nnd an OS which is installed in a network 
computer such ^s J^vaOS as well as a chip card OS 
as described m the nbove without being limited to any 
specific opcrn'inQ system environment. 

The opcfriirng svslom in the supporting server side 
240 may bo impicnxjnled by one which supports GUI 
multi-window environment as a standard, such as Win- 
dows NT (d I'rtdcmrtrk of Microsoft). Windows 95 (a 
trademark ol Microsolt}. Windows 3.x (a trademark of 
Microsoft). OS *2 in trademark of IBM), MacOS (a trade- 
mark of Apple) r.nd X-WINDOW system (a trademark 
of MIT) on AIX (rf trademark of IBM), a real time OS. 
such as OS/Oocn (a trademark of IBM) and VxWorks(a 
trademark of V^md R vcr Systems. Inc) and an OS which 
is installed m a network computer, such as JavaOS with- 
out being limited :o any specific operating system envi- 
ronment. 

Fig. 2 is a diagram illustrating the operational condi- 
tion of the system in a prelerred embodiment of this in- 
vention. In this embodiment, the portable information 
terminal 210 has an access to the supporting server 240 
by a PHS function 

A connection request is sent from the portable in- 
fornnation terminal 210 to the supporting server 240. 

The supporting server 240 receives the connection 
request and notilies the portable information terminal 
210 that it can be connected. Fig.2 shows a state in 
which a remote controlling application on the portable 
information terminal 210 and a remote controlled appli- 
cation on the supporting server 240 has established a 



connection. 

In the figure, "PIFS* is an abbreviation of "PHS In- 
ternet Access Forum Standard* which is a communica- 
tion method for a high speed wireless communication at 
5 32 Kbps with PHS. 

Fig.3 IS a functional block diagram explaining the 
system configuration of the portable information termi- 
nal 210 and the supporting server 240 in the preferred 
embodiment of this invention. In the figure, portable in- 

10 formation terminal 210 side includes an input informa- 
tion monitoring part 303, an input information sending 
part 307. a rendering instruction receiving part 333. a 
rendering instruction storing pari 335, a rendering in- 
struction analysing part 337 and a togical screen image 

IS data storing part 339. 

On the other hand, the supporting server 240 side 
includes an input information receiving part 311 , an input 
information storing part 313. an input information ana- 
lysing part 315. a window system 320, a rendering in- 

20 struction storing part 327 and a rendering instruction 
sending part 329. 

The input information generating part 303 gener- 
ates input information to be described later based on a 
key input taking place by operating an input device such 

25 as a keyboard on the portable information terminal 210. 
The input information sending part 307 sends the input 
information to the supporting server 240, 

The input information receiving part 31 1 receives in- 
put information transmitted from the portable informa- 

30 tion terminal 210 side and stores the input information 
in the input information storing part 31 3 in the supporting 
server 240 side to be described later. The input informa- 
tion storing part 31 3 temporarily stores input information 
to make the Input information receiving part 311 and the 

35 input information analysing part 315 Independently op- 
erable. 

The input information analysing part 315 monitors 
the input information to be stored in the input information 
storing part 31 3 of the supporting server side and anal- 

40 yses the input information stored In the input information 
storing part 313. The input information analysing part 
315 then converts it to a message (window message) in 
the form conforming to the window system installed in 
the supporting server 240 lor delivery to the window sys- 

45 tem 320. 

In the preferred embodiment of this Invention, the 
window system includes a rendering Instruction analys- 
ing part 321 . a rendering Instruction monitoring part 323 
and a rendering driver 325. 

so Re-writing of the GUI screen caused by the window 
message Is outputted as an image to a display device 
245 by an image rendering engine 321 and the render- 
ing driver 325 of the window svstem 320. The rendering 
instruction monitoring part 323 monitors all APIs called 

55 for re-wrlting the GUI screen of the supporting server, 
prepares a rendering Instruction based on this, and per- 
forms a conversion processing of Image data to be de- 
scribed later between the image rendering engine and 
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the rendering driver The rendering instruction monitor- 
ing part 323 atso stores a rendering instruction including 
the converted inrege data in the rendering instruction 
storing pari 327 of the supporting server 240 side. 

The rendering instruction storing part 313 tennpo- 
rarily stores a rendering instruction to make the render- 
ing instruction monitoring part 323 and the rendering in- 
struction sending part 329 independently operable. 

The rendering instruction sending part 329 monitors 
to see that a rendering instruction is stored in the ren- 
dering instruction storing part 327 and, when detecting 
that a rendering instruction is stored in the rendering in- 
struction storing part 327. sends the rendering instruc- 
tion to the portable information terminal 210 side. 

The rendering instruction receiving pari 333 moni- 
tor s transmission of a rendering instruction from the sup- 
porting server and stores the received rendering instruc- 
\*on in the rendering instruction storing part 335 of the 
poftrtOlo information terminal 210 side. 

Trie rendering instruction analysing part 327 moni- 
luib lu bce that a rendering instruction is stored in the 
rcoOormg instruction storing part 335 and composes an 
irrvigc v^hich is resulted from image conversion process 
to t>c described later applied to the image generated in 
ihc window system of the supporting server by analys- 
ing tro Elorod rendering instruction into a logical screen 
irrvigo of bit map. By monitoring the change of the 
screen by a software timer or a window message hook 
Hnd gonoraling a rendering instruction from the differ- 
ence H similar system can be constructed though the 
operating speed is somehow sacrificed. 

A display region specifying part 341 specifies a re- 
gion in the screen 211 of the portable information termi- 
nnl 210 from the bgical screen image of the bit map for 
output to the display screen 211. 

While each functional block shown in Fig.3 has 
been described in the above, it is a logical functional 
block and is not meant to be implemented in the form of 
a dtscroie hardware or a software. It can be implement- 
ed by a shared hardware or software. Also, it is not 
meant that all functional blocks shown in Fi9.3 are in- 
dispensable components of this invention. For example, 
when the supporting server 240 is customised to sup- 
port a remote controlling terminal, the rendering driver 
325 or the display screen 245 is a component which is 
not needed in this invention because it does not have to 
perform screen display by itself. 

Next, a procedure of the remote control in the pre- 
ferred embodiment of this invention will be described 
with reference to the flow chart of Fig.4 to 1 0 and a sche- 
matic diagram of Fig 11 showing the operational condi- 
tion of the portable information terminal 21 0 and the sup- 
porting server 240. Fig.4 and Fig.5 are flow chart show- 
ing the procedure of processing in the portable informa- 
tion terminal 210 side in the preferred embodiment of 
this invention. 

The operator starts up the portable information ter- 
minal 210 (block 401) and sends out a connection re- 



quest to the supponing server 240 using the PHS func- 
tion of the portable inlornrwtion terminal 2lO (block 403). 
When the connection is successful (block 405) and the 
supporting server 240 is ready (block 411 ). information 

5 indicating the processing capability of the portable infor- 
mation terminal is sent from the portable information ter- 
minal 21 0 to the supporting server 240 side (block 413). 
In the preferred embodiment of this invention, the 
processing capability information includes the resolu- 

10 tion of the logical screen, the display capability of colour 
image (number of bits information per picture element 
such as 256 colours. 1 6 colours and monochrome), col- 
our palette information and compression/decompres- 
sion capability information, etc. 

J5 When the rendering instruction can be received 
(block 415). a rendering instruction of an initial screen 
is received from the supporting server 240 (block 417) 
and the received rendering instruction is stored in the 
rendering instruction storing part 335 of the portable in- 

20 formation terminal 210 (bkx:k 419). 

When reception of the rendering instruction has 
been completed (block 421), the rendering instruction 
analysing part 337 analyses the rendering instruction 
stored in the rendering instruction storing part 335 and 

2S generates an image which is resulted from an image 
conversion applied to the image rendered on the GUI 
screen of the supporting server for rendering on the 
screen 211 of the portable information terminal 210 
(block 425). 

30 In the preferred embodiment of this invention, a cer- 
tain region is cut out of the logical screen 213 on the 
memory as a display region for display as shown in Fig. 
11. The will be described later in this specification. 
The input information generating part 303 monitors 

35 input information taking place by a key input on the key- 
board of the portable information terminal 210 (block 
426) and converts the key input to a mouse key input 
(blocks 427, 429). 

In the preferred embodiment of this invention, the 

^0 signal generated by the key input is converted by the 
input information generating part 303 to input infornria- 
tion such as the type of input information and the coor- 
dinate position where the Input information is generated 
tor delivery to the input information sending part 307 as 

45 shown in Fig. 12. A sample of input information in the 
preferred embodiment of this invention is shown in Fig, 
13. The input information sending part 307 sends this 
input information to the supporting server 240 side. 
The input information may be sent to the supporting 

so server 240 with compression or encryption and then de- 
compressed or de-encrypted for use in the supporting 
server 240 side. 

When the logical screen is updated or the mouse Is 
moved in the logical screen (blocks 435, 437). the dis- 

55 play region specifying part 341 calculates a screen off- 
set position (block 439) and renders a part of the logical 
screen specified by the calculation in the display screen 
21 1 (block 441 ). Also, the mouse pointers are composed 
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(block 443). 

In the preferred embodiment of this invention, the 
size of the region 215 to be cut out as a display region 
is determined from the size of the logical screen 21 3 on 
the memory, the size of the display screen 211 of the 5 
portable information terminal 210 and position informa- 
tion of the mouse pointer as shown in Fig. 11. 

Specifically, by putting the start point (upper left co- 
ordinate position) of the cut out region 215 as (ox. oy), 
the size of the logical screen 21 3 on the memory as (sx, io 
sy). the size of the display screen 211 of the portable 
information terminal 21 0 as (wx. wy) and the coordinate 
position of the mouse pointer as (px. py), 



ox and oy are determined by: 



ox = px (sx - wx)/(sx - 1) 



oy = py (sy-wy)/(sy-1) 
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20 



and a region (wx, wy) having start points ox and oy is 
determined to be a cut out region 215. However, this is 
a mere example and the position of the mouse pointer 2S 
may be always at the centre (excluding the periphery) 
of the display screen. 

In this case, ox and oy are determined by; 



30 



35 



40 



ox = px - wx/2 

oy = py - wy/2 

(provided ox = 0 when ox < 0 
ox = sx - wx when ox > sx - wx 
oy = 0 when oy < 0 
oy = sy - wy when oy > sy - wy) 



and a region having ox and oy is determined to be the 
cut out region 215. 45 

Fig.6 to Fig. 1 0 are flow charts showing the process- 
ing procedure ol the supporting server 240 side in the 
preferred embodiment of this invention. 

When the supporting server 240 is started (block 
501 ), the supporting server 240 determines whether or so 
not there is a connection request from the portable in- 
formation terminal (block 503). 

When there is a connection request, the shell pro- 
gram is started to start the logical window system (block 
507), In the preferred embodiment of this invention, the 55 
supporting sen/er 240 can run a plurality of shell proc- 
esses and enables connection to a plurality of portable 
information terminals to allow a parallel processing of 



shell processes. 

■logical window sysiem* as used here means a win- 
dow system in which an input from a physical mouse or 
keyboard is neglected and output to a physical video de- 
vice is disabled. Because such logical window system 
does not access to a physical device, a plurality of such 
systems can be active simultaneously. 

Then, processing capability information sent from 
the portable information terminal 210 is received (block 
509). and the rendering instructbn monitoring part 323, 
the rendering instruction sending part 329. the input in- 
formation receiving part 311 and the input information 
analysing part 315 are each activated (blocks 511. 513. 
514.515), 

In the preferred embodiment of this invention, when 
the supporting server 240 starts, it re-renders the GUI 
screen of the supporting server 240 (bkx;k 512) and 
sends it to the portable information terminal 210 as a 
rendering instruction. 

When the end of operation is delected (block 516), 
the logical window system and the shell process are 
ended (block 517. 518). 

Fig.7 is a flow chart showing a processing proce- 
dure of the rendering instruction monitoring part 323. 
The rendering instruction monitoring part 323 inter- 
venes between the GUI of the window system 320 and 
the rendering driver 325 and can behave as if it is the 
rendering driver 325 relative to the image rendering en- 
gine 321 while it behaves as if it is the image rendering 
engine 321 relative to the rendering driver 325. 

The rendering instruction monitoring part 323 mon- 
itors between the image rendering engine and the ren- 
dering driver, monitors all API's called for re-writing the 
GUI screen of the supporting server (block 521), pre- 
pares a rendering instruction based on this (block 523), 
and stores it in the rendering instruction storing part 327 
of the supporting server 240 after image conversion 
(block 525, 527). Fig. 1 5 is an example of data of a typical 
rendering instruction in the preferred embodiment of this 
invention. Incidentally, it the resolution of the output de- 
vice of the portable inlormation terminal is sufficiently 
high, an output may be directly provided to the output 
device of the portable information terminal by omitting 
the process of rendering lor the logical screen. 

In the preferred embodiment of this invention, an 
image conversion is performed according to the 
processing capability specifying information sent from 
the portable information terminal 210. In the example of 
Fig. 1 1 , the portable information terminal 210 notifies the 
supporting server 240 that an image of two gradations 
of white and black is handled. 

Fig. 16 shows a grey scale conversion from a colour 
image to multi gradations of white and black while Fig. 
17 shows a dithering conversion from multi gradations 
of white and black to two gradations of white and black. 
The dithering conversion may be replaced by optimising 
with the error diffusion method known in the art. By per- 
forming various conversions which the portable informa- 
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lion terminal 210 requires in the supporting server 240 
side the load of the portable information terminal 210 
Cfin be reduced and the communication 'volume be- 
tvvccn the server 240 and the portable information ter- 
minnt 210 can be reduced. 5 

The converted rendering instruction may be further 
compressed cr encrypted according to the processing 
capability specifying information sent from the portable 
infofmntion terminal 210 and sent to the portable infor- 
mrtlion terminal 21 0 where it may be decompressed for io 
use 

Fig 8 IS a flow chart showing a processing proce- 
dure of the rendering instruction sending part 329. The 
rendering instruction sending part 329 monitors to see 
that a rendering instruction is stored in the rendering in- i5 
struction storing part 327 (block 535). When it is detect- 
ed that the (crdCMng instruction has been stored in the 
rendering tnsiruciton storing part 327, the rendering in- 
struction scnamg part 329 sends the rendering instruc- 
tion ID trie poiirtbic inlormdiion terminal 2i0side (block 20 
537) Tfiii piuceduie is continued until the end of 
process ng (btock 537) and. upon detection of the end 
of process ng ihc rendering instruction sending part 
329 IS devicitv.itcd (block 541). 

Fig 0 ic .1 lio^ Chan showing a processing proce- 2S 
dure of the input infcrmation receiving part 311. The in- 
put information receiving part 31 1 , upon receiving oper- 
ation sirirt input information from the portable informa- 
tion terminal 21 0 determines that input information can 
be received it>\oc^ 551 ). receives input information sent 50 
from the portable information terminal 210 side (block 
553) and stores tic input information in the input infor- 
mation stor :ng pHl 31 3 of the supporting server 240 side 
(block 555) In input information receiving part 311 also, 
the procedure is continued until the end of operation 35 
(block 557) and upon detection of the end of operation, 
the input infomalion receiving part 311 is deactivated 
(block 459) 

The input mlormation analysing part 315 monitors 
to see that input information is stored in the input infor- ^0 
mation storing part 313 as shown in Fig.lO (block 561) 
and analyses the input information stored in the input 
information storing part 31 3 (block 563). In the preferred 
embodiment ci tris invention, the input information an- 
alysing part 31 5 analyses the received input information ^5 
and converts jI to a message of a form conforming to 
the window system installed in the supporting server for 
delivery to the window system 320 via a journal playback 
hook 307 (block 571). 

The journal playback hook 397 is usually used to- so 
gether with a journal record hook. The journal record 
hook can record a window message generated by an 
operator input and reproduce the recorded window mes- 
sage by the journal playback hook. 

In the preferred embodiment of this invention, the 55 
function of the journal playback hook is utilised. By de- 
livering a window message generated in the input infor- 
mation analysing part to the journal playback hook 307, 



the window svstem 320 can handle it as a window mes- 
sage generated within the window system. The window 
message delivered to the window system 320 is inter- 
preted by the window system 320 to determine what in- 
put was given to what application and causes the GUI 
screen to be re-written. This re-writing of the GUI screen 
caused by the window message is outputted to the dis- 
play device 245 as an image by the image rendering 
engine 321 and the rendering driver 325 of the window 
svstem 320. 

In the input information analysing part also, this pro- 
cedure is continued until the end of operation (block 
565) and, upon detection of the end of operation, the 
input information analysing part 315 notifies each part 
of the end of operation (block 567) and is then deacti- 
vated (block 569). 

As described in the above, according to this inven- 
tion, a sysiem is provided which is capable of executing 
a general purpose OS as used in a conventional PC and 
a high level application operating on such OS in a ma- 
chine having limited resources. 

In one mode of this invention, a small size portable 
terminal of a low cost and providing a high function is 
provided. 

In one mode of this invention, a portable information 
terminal is provided which can execute an application 
used in home or office. 

In one mode of this invention, a portable information 
terminal having a low probability of losing data is pro- 
vided. 

In one mode of this invention, it is possible to reduce 
the work of installing a remote control software which 
was generated in the side of remote controlling terminal 
for enabling a consolidated control of a system. 

In one mode of this invention, a system is provided 
which is not affected by the type of a remote controlling 
terminal for realising a common operability. 

In one mode of this invention, it is possible to reduce 
the usage of resources of a remote controlling terminal 
for reducing the hardware requirement to enable a gen- 
eral purpose OS and a general purpose application lo 
be used. 

In one mode of this invention, a high speed remote 
control system is provided which reduces the volume of 
communication data between a remote controlling ma- 
chine and a remote controlled machine in remote control 
operation. 

In one mode of this invention, a remote control 
svstem is provided in which data is transmitted in a form 
optimum to a remote controlling machine in performing 
a remote control. 

In one mode of this invention, a remote control sys- 
tem is provided which can simultaneously support a plu- 
rality of remote controlling machines in performing a re- 
mo\e control. 
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Claims 

1 . A method or any one of system, a sen/er (247) or a 
recording medium (20) having a program code for 
instructing said server for remote controlling re- 
sources retained by a server existing on a network 
Irom a terminal (215) which has an input device 
(301) and a display device (220). comprising; 

(a) receiving or means for receiving (311) or in- 
structing receiving respectively the input infor- 
mation transmitted from said terminal side 
(215). 

(b) analysing or means for analysing (315) or 
instructing analysing respectively the input in- 
tormalion. 



respectively a processing capability specifying in- 
formation transmitted from said terminal (215). 

5. A method, system, server (247) or recording medi- 
s um (20) having a program code for instructing said 
server as claimed in any previous claim in which the 
processing capability specifying information is ei- 
ther one of; 

10 (a) maximum resolution information of a logical 

screen, 

(b) maximum resolution information of a phys- 
ical screen, 

(c) bits per picture element information, 
IS (d) colour palette informatbn, 

(e) compression/decompression capability in- 
formation, of said terminal. 



(c ) gcnc/ating or means for generating (320) or 
iniiruciing gcneraling respectively first render- 20 
irio iMMyc Ceiia consisting of a first number of 
bits pot picture element based on the content 
ol :hc rtfVilyscd input information, 

(di converting or means for converting or in- 2S 
str jci ng converting respectively said first ren- 
do'ing imngo data to second rendering image 
da:H consisting of a second number of bits per 
pif luro riomont less than the first number of 
bits rtnd 30 

(e) trrtnsmiiiing or means for transmitting (329) 
or instructing transmitting respectively a ren- 
dering tnsiruction including said second render- 
ing irTV4gc Irom said server side (247) to said 3S 
lerminHl skJg (215). 



6. A method for specifying a display region of a size 
of (wx. wy) to be displayed on a display screen from 
image data constituting a logical screen having a 
size of (sx, sy) stored in a portable information ter- 
minal, characterised in that the region of the size of 
(wx, wy) is specified as a display region starting 
from a starting point which is defined as; 

ox - px(sx - wx)/((sx - 1) 

oy = py(sy-wy)/(sy- 1) 

where (px. py) is a coordinate position of a mouse 
pointer of the portable information terminal. 



2. A method system server or recording medium as 
claimed m clnim 1 including analysing or means for 
analysing (337) or analysing respectively the ren- 40 
dering instruction transmitted from the server side 
on said terminal side (215). and 
generating or means for generating or generating 
respectively tmagc data to be displayed in a display 
screen (220) ol said terminal (215) according to the 
content ol the analysed rendering instruction (337). 



3. A method system server or recording medium or a 
program code lespectively as claimed in claim 1 In- 
cluding starting or means for starting (505) respec- ^0 
tively a shell process in response to a connection 
request from a first terminal of a plurality of termi- 
nals. 



4. A method system, server (247) or recording medi- 55 
um (20) as claimed in any of claims 1 to 3 and in- 
cluding: 

receiving or means for receiving (311) or receiving 
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